Touch-Sensitive Remote Control with Visual Feedback

ABSTRACT

An audio/video (A/V) hub provides feedback to a user of a portable electronic device with a touch-sensitive display (such as a cellular telephone) that is used as a wireless remote control for an audio/video (A/V) display device, the A/V hub and/or a consumer-electronic device. In particular, when the A/V hub receives, from the portable electronic device, user-interface activity information associated with a user interface displayed on a touch-sensitive display, the A/V hub generates visual feedback based on the user-interface activity information. Then, the A/V hub provides the visual feedback to the A/V display device for display on the A/V display device. The visual feedback indicates a position of at least a touch contact point of a user of the portable electronic device relative to a strike area of at least a virtual command icon in the user interface.

BACKGROUND

Field

The described embodiments relate to techniques for providing feedback toa user of a portable electronic device, including providing visualfeedback to a user about their interaction with a user interface on atouch-sensitive display in a portable electronic device.

Related Art

The versatility and capabilities of portable electronic devices isincreasing their popularity. For example, many portable electronicdevices includes touch-sensitive displays that allow users todynamically interact with the portable electronic devices. In addition,many portable electronic devices can wirelessly communicate with otherelectronic devices, which allow the portable electronic devices torapidly and conveniently communicate information. In particular, theportable electronic devices may include networking subsystem thatimplement a network interface, such as: a wireless network described inthe Institute of Electrical and Electronics Engineers (IEEE) 802.11standard, Bluetooth® (from the Bluetooth Special Interest Group ofKirkland, Wash.), and/or another type of wireless network. Thecombination of a user interface on a touch-sensitive display andwireless-communication capability can allow users to use portableelectronic devices to remotely control another electronic device.

However, it can be difficult to use a user interface on atouch-sensitive display. For example, many touch-sensitive displays donot provide tactile or non-visual sensory feedback to a user of aportable electronic device. Consequently, the user may need to look ator directly view the user interface on the touch-sensitive displays inorder to correctly activate a strike area of a given virtual icon in theuser interface. When this is not possible, the user may make errors,such as activating the wrong strike area or missing a desired strikearea. These errors will frustrate users, which can degrade the userexperience.

SUMMARY

The described embodiments include an audio/video (A/V) hub. This A/V hubincludes: an antenna; an interface circuit, coupled to the antenna,which communicates with a portable electronic device; and a controlcircuit coupled to the interface circuit. During operation, the controlcircuit receives, via the interface circuit, user-interface activityinformation from the portable electronic device, where theuser-interface activity information is associated with a user interfacedisplayed on a touch-sensitive display in the portable electronicdevice, and the user interface is used to control an A/V display device,the A/V hub and/or a consumer-electronic device, which is other than theportable electronic device. Then, the control circuit generates visualfeedback based on the user-interface activity information. Next, thecontrol circuit provides the visual feedback to the A/V display devicefor display on a display in the A/V display device, where the visualfeedback indicates a position of at least a touch contact point of auser of the portable electronic device relative to a strike area of atleast a virtual command icon in the user interface. (In someembodiments, the control circuit displays the visual feedback on thedisplay.)

Moreover, the interface circuit may communicate with the A/V displaydevice. Furthermore, the A/V hub may include a second interface circuit,coupled to the processor, which can couple to the A/V display device.For example, the A/V hub may include a port, coupled to the secondinterface circuit, which is compatible with a high-definitionmultimedia-interface (HDMI) standard, and the A/V hub may couple to theA/V display device via the port.

Additionally, the visual feedback may be displayed on the display alongwith A/V content. Note that the visual feedback may be superimposed onthe A/V content, and the visual feedback may be partially transparent sothat the A/V content is visible underneath the visual feedback whendisplayed on the A/V display device.

In some embodiments, the control circuit: generates the A/V content; andprovides the A/V content to the A/V display device.

Note that the A/V display device may include a television. Moreover, theportable electronic device may include a cellular telephone thatfunctions as a remote control.

Furthermore, at least the touch contact point of the user is ongoing(i.e., the user may maintain contact with the touch-sensitive display).

Additionally, the user-interface activity information may specifyactivation of at least the virtual command icon based on the userreleasing at least the touch contact point within the strike area.

In some embodiments, the control circuit includes: a processor coupledto the interface circuit; and a memory, coupled to the processor, whichstores a program module that is executed by the processor. The programmodule may include instructions for: the receiving; the generating; andthe providing.

Another embodiment provides a computer-program product for use with theA/V hub. This computer-program product includes instructions for atleast some of the operations performed by the A/V hub.

Another embodiment provides a method for providing visual feedback. Thismethod includes at least some of the operations performed by the A/Vhub.

Another embodiment provides the portable electronic device.

Another embodiment provides a second computer-program product for usewith the portable electronic device. The second computer-program productincludes instructions for at least some of the operations performed bythe portable electronic device.

Another embodiment provides a method for providing user-activityinformation. This method includes at least some of the operationsperformed by the portable electronic device.

This Summary is provided merely for purposes of illustrating someexemplary embodiments, so as to provide a basic understanding of someaspects of the subject matter described herein. Accordingly, it will beappreciated that the above-described features are merely examples andshould not be construed to narrow the scope or spirit of the subjectmatter described herein in any way. Other features, aspects, andadvantages of the subject matter described herein will become apparentfrom the following Detailed Description, Figures, and Claims.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram illustrating a system with electronic deviceswirelessly communicating in accordance with an embodiment of the presentdisclosure.

FIG. 2 is a flow diagram illustrating a method for providing visualfeedback in accordance with an embodiment of the present disclosure.

FIG. 3 is a block diagram illustrating visual feedback on an audio/video(A/V) display device in accordance with an embodiment of the presentdisclosure.

FIG. 4 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 5 is a block diagram illustrating one of the electronic devices ofFIG. 1 in accordance with an embodiment of the present disclosure.

Note that like reference numerals refer to corresponding partsthroughout the drawings. Moreover, multiple instances of the same partare designated by a common prefix separated from an instance number by adash.

DETAILED DESCRIPTION

An audio/video (A/V) hub provides feedback to a user of a portableelectronic device with a touch-sensitive display (such as a cellulartelephone) that is used as a wireless remote control for an A/V displaydevice, the A/V hub and/or a consumer-electronic device. In particular,when the A/V hub receives, from the portable electronic device,user-interface activity information associated with a user interfacedisplayed on a touch-sensitive display, the A/V hub generates visualfeedback based on the user-interface activity information. Then, the A/Vhub provides the visual feedback to the A/V display device for displayon the A/V display device. The visual feedback indicates a position ofat least a touch contact point of a user of the portable electronicdevice relative to a strike area of at least a virtual command icon inthe user interface.

By providing the visual feedback, this feedback technique facilitatesintuitive use of the user interface displayed on the touch-sensitivedisplay. In particular, even when the user is not directly looking atthe user interface displayed on the touch-sensitive display, the visualfeedback allows the user to navigate through and use the user interface(i.e., the visual feedback provides spatial situation awareness to theuser). This capability allows the user to effectively use the userinterface to control the A/V hub, the A/V display device and/or theconsumer-electronic device with fewer errors or mistakes, even when theuser interface is obstructed (such as when the portable electronicdevice is under a blanket) or when lighting conditions are poor, andeven though there may not be tactile or haptic clues provided on asurface of the user interface to guide the user's touch-contact point tothe strike area. Consequently, the feedback technique may improve theuser experience when using the portable electronic device, the A/V huband the A/V display device.

In the discussion that follows the portable electronic device, the A/Vhub and/or the A/V display device may include radios that communicatepackets or frames in accordance with one or more communicationprotocols, such as: an Institute of Electrical and Electronics Engineers(IEEE) 802.11 standard (which is sometimes referred to as ‘Wi-Fi®,’ fromthe Wi-Fi® Alliance of Austin, Tex.), Bluetooth® (from the BluetoothSpecial Interest Group of Kirkland, Wash.), a cellular-telephonecommunication protocol, a near-field-communication standard orspecification (from the NFC Forum of Wakefield, Mass.), and/or anothertype of wireless interface. In the discussion that follows, Wi-Fi isused as an illustrative example. For example, the cellular-telephonecommunication protocol may include or may be compatible with: a 2^(nd)generation of mobile telecommunication technology, a 3^(rd) generationof mobile telecommunications technology (such as a communicationprotocol that complies with the International MobileTelecommunications-2000 specifications by the InternationalTelecommunication Union of Geneva, Switzerland), a 4^(th) generation ofmobile telecommunications technology (such as a communication protocolthat complies with the International Mobile Telecommunications Advancedspecification by the International Telecommunication Union of Geneva,Switzerland), and/or another cellular-telephone communication technique.In some embodiments, the communication protocol includes Long TermEvolution or LTE. However, a wide variety of communication protocols maybe used. In addition, the communication may occur via a wide variety offrequency bands. Note that the portable electronic device, the A/V huband/or the A/V display device may communicate using infra-redcommunication that is compatible with an infra-red communicationstandard.

Communication among electronic devices is shown in FIG. 1, whichpresents a block diagram illustrating a system 100 with a portableelectronic device 110 (such as a remote control or a cellulartelephone), an A/V hub 112, A/V display device 114 (such as atelevision, a monitor, a computer and, more generally, a displayassociated with an electronic device) and one or moreconsumer-electronic devices 116 (e.g., a radio receiver, a video player,a satellite receiver, an access point that provides a connection to awired network such as the Internet, a media or a content source, aconsumer-electronic device, a set-top box, over-the-top contentdelivered over the Internet or a network without involvement of a cable,satellite or multiple-system operator, etc.). (Note that A/V hub 112,A/V display device 114, and the one or more consumer-electronic devices116 are sometimes collectively referred to as ‘components’ in system100) In particular, portable electronic device 110 and A/V hub 112 maycommunicate with each other using wireless communication, and A/V hub112 and other components in system 100 (such as A/V display device 114and the one or more consumer-electronic devices 116) may communicateusing wireless and/or wired communication. (Thus, portable electronicdevice 110 may communicate with A/V display device 114 and the one ormore consumer-electronic devices 116 that may be remotely located, suchas in another room or that are not within line of sight, and which mayor may not be connected by high-definition multimedia-interface (HDMI)cable 122.) During the wireless communication, these electronic devicesmay wirelessly communicate while: transmitting advertising frames onwireless channels, detecting one another by scanning wireless channels,establishing connections (for example, by transmitting associationrequests), and/or transmitting and receiving packets or frames (whichmay include the association requests and/or additional information aspayloads, such as user-interface information, device-state information,user-interface activity information, data, A/V content, etc.).

As described further below with reference to FIG. 5, portable electronicdevice 110, A/V hub 112. A/V display device 114 and the one or moreconsumer-electronic devices 116 may include subsystems, such as: anetworking subsystem, a memory subsystem and a processor subsystem. Inaddition, portable electronic device 110 and A/V hub 112, and optionallyone or more of Ai/V display device 114 and/or the one or moreconsumer-electronic devices 116, may include radios 118 in thenetworking subsystems. (Note that radios 118 may be instances of thesame radio or may be different from each other.) More generally,portable electronic device 110 and A/V hub 112 (and optionally one ormore of A/V display device 114 and/or the one or moreconsumer-electronic devices 116) can include (or can be included within)any electronic devices with the networking subsystems that enableportable electronic device 110 and A/V hub 112 (and optionally one ormore of A/V display device 114 and/or the one or moreconsumer-electronic devices 116) to wirelessly communicate with eachother. This wireless communication can comprise transmittingadvertisements on wireless channels to enable electronic devices to makeinitial contact or detect each other, followed by exchanging subsequentdata/management frames (such as association requests and responses) toestablish a connection, configure security options (e.g., InternetProtocol Security), transmit and receive packets or frames via theconnection, etc.

As can be seen in FIG. 1, wireless signals 120 (represented by a jaggedline) are transmitted from radio 118-1 in portable electronic device110. These wireless signals are received by at least A/V hub 112. Inparticular, portable electronic device 110 may transmit packets. Inturn, these packets may be received by a radio 118-2 in A/V hub 112.This may allow portable electronic device 110 to communicate informationto A/V hub 112. While FIG. 1 illustrates portable electronic device 110transmitting packets, note that portable electronic device 110 may alsoreceive packets from A/V hub 112.

In the described embodiments, processing of a packet or frame inportable electronic device 110 and A/V hub 112 (and optionally one ormore of A/V display device 114 and/or the one or moreconsumer-electronic devices 116) includes: receiving wireless signals120 with the packet or frame; decoding/extracting the packet or framefrom received wireless signals 120 to acquire the packet or frame; andprocessing the packet or frame to determine information contained in thepacket or frame (such as the information associated with a data stream).For example, the information from portable electronic device 110 mayinclude user-interface activity information associated with a userinterface displayed on touch-sensitive display 124 in portableelectronic device 110, which a user of portable electronic device 110uses to control A/V hub 112, A/V display device 114 and/or one of theone or more consumer-electronic devices 116. Alternatively, theinformation from A/V hub 112 may include device-state information abouta current device state of A/V display device 114 or one of the one ormore consumer-electronic devices 116 (such as on, off, play, rewind,fast forward, a selected channel, selected content, a content source,etc.), or may include user-interface information for the user interface(which may be dynamically updated based on the device-state informationand/or the user-interface activity information). (Note that A/V hub 112may determine the device-state information even for legacy entertainmentdevices that are only capable of receiving commands or instructions,i.e., that are only capable of unidirectional communication.)Furthermore, the information from A/V hub 112 and/or one of the one ormore consumer-electronic devices 116 may include audio and video thatare displayed on A/V display device 114. (However, as noted previously,the audio and video may be communicated between components in system 100via wired communication. Therefore, as shown in FIG. 1, there may be awired cable or link, such as HDMI cable 122, between A/V hub 112 and A/Vdisplay device 114.)

Note that the communication between portable electronic device 110 andA/V hub 112 (and optionally one or more of A/V display device 114 and/orthe one or more consumer-electronic devices 116) may be characterized bya variety of performance metrics, such as: a data rate, a data rate forsuccessful communication (which is sometimes referred to as a‘throughput’), an error rate (such as a retry or resend rate), amean-square error of equalized signals relative to an equalizationtarget, intersymbol interference, multipath interference, asignal-to-noise ratio, a width of an eye pattern, a ratio of number ofbytes successfully communicated during a time interval (such as 1-10 s)to an estimated maximum number of bytes that can be communicated in thetime interval (the latter of which is sometimes referred to as the‘capacity’ of a channel or link), and/or a ratio of an actual data rateto an estimated data rate (which is sometimes referred to as‘utilization’). Moreover, the performance during the communicationassociated with different channels may be monitored individually orjointly (e.g., to identify dropped packets).

The communication between portable electronic device 110 and A/V hub 112(and optionally one or more of A/V display device 114 and/or the one ormore consumer-electronic devices 116) in FIG. 1 may involve one or moreindependent, concurrent data streams in different wireless channels (oreven different Wi-Fi communication protocols) in one or more connectionsor links, which may be communicated using multiple radios. Note that theone or more connections or links may each have a separate or differentservice set identifier on a wireless network in system 100 (which may bea proprietary network or a public network). Moreover, the one or moreconcurrent data streams may, on a dynamic or packet-by-packet basis, bepartially or completely redundant to improve or maintain the performancemetrics even when there are transient changes (such as interference,changes in the amount of information that needs to be communicated,movement of portable electronic device 110, etc.), and to facilitateservices (while remaining compatible with the Wi-Fi communicationprotocol) such as: channel calibration, determining of one or moreperformance metrics, performing quality-of-service characterizationwithout disrupting the communication (such as performing channelestimation, determining link quality, performing channel calibrationand/or performing spectral analysis associated with at least onechannel), seamless handoff between different wireless channels,coordinated communication between components, etc. These features mayreduce the number of packets that are resent, and, thus, may decreasethe latency and avoid disruption of the communication and may enhancethe experience of one or more users or viewers of content on A/V displaydevice 114.

As noted previously, a user may control A/V hub 112, A/V display device114 and/or one of the one or more consumer-electronic devices 116 viathe user interface displayed on touch-sensitive display 124 on portableelectronic device. In particular, at a given time, the user interfacemay include one or more virtual icons that allow the user to activate,deactivate or change functionality or capabilities of A/V hub 112, A/Vdisplay device 114 and/or one of or more consumer-electronic devices116. For example, a given virtual icon in the user interface may have anassociated strike area on a surface of touch-sensitive display 124. Ifthe user makes and then breaks contact with the surface (e.g., using oneor more fingers or digits, or using a stylus) within the strike area,portable electronic device 110 (such as a processor executing a programmodule) may receive user-interface activity information indicating thiscommand or instruction from a touch-screen input/output (I/O)controller, which is coupled to touch-sensitive display 124.(Alternatively, touch-sensitive display 124 may be responsive topressure. In these embodiments, the user may maintain contact withtouch-sensitive display 124 with an average contact pressure that isusually less than a threshold value, such as 10-20 kPa, and may activatea given virtual icon by increase the average contact pressure withtouch-sensitive display 124 above the threshold value.) In response, theprogram module may instruct an interface circuit in portable electronicdevice 110 to wirelessly communicate the user-interface activityinformation indicating this command or instruction to A/V hub 112, andA/V hub 112 may communicate the command or the instruction to the targetcomponent in system 100 (such as A/V display device 114). Thisinstruction or command may result in A/V display device 114 turning onor off, displaying content from a particular source, performing a trickmode of operation (such as fast forward or rewind), etc.

However, one problem with using portable electronic device 110 as aremote control for another component in system 100 is that the surfaceof touch-sensitive display 124 may lack tactile or haptic clues (such asroughness or changes in the topography) that the user can use tonavigate through the virtual icons in the user interface just bytouching touch-sensitive display 124 (e.g., even when the user is notlooking at the user interface or when the user cannot look at the userinterface, such as when the lighting is poor or when portable electronicdevice 110 is hidden under a blanket, etc.). This is in contrast withmany existing remote controls, which have physical buttons and featuresthat allow the user to use haptic perception to feel when their fingeris above a button or, even, to feel when their finger is above aparticular button. Consequently, when using the user interface onportable electronic device 110, the user may be at risk for positioningtheir finger or digit (or a stylus) over the wrong virtual icon, or theuser may be at risk for mis-positioning their finger or digit (or astylus) relative to the strike area of a given virtual icon. Theseerrors or mistakes may frustrate the user, thereby degrading their userexperience when using components in system 100, such as A/V hub 112, A/Vdisplay device 114 and/or one of the one or more consumer-electronicdevices 116.

In order to address this problem, after receiving the user-interfaceactivity information (such as information specifying the location of acurrent, ongoing touch contact point of the user or a stylus with thesurface of touch-sensitive display 124), A/V hub 112 (such as aprocessor executing a program module) may generate visual feedback basedon the user-interface activity information. Then, A/V hub 112 mayprovide, via an interface circuit in A/V hub 112, visual feedback to A/Vdisplay device 114 for display on a display in A/V display device 114.(In some embodiments, A/V hub 112 displays the visual feedback on A/Vdisplay device 114.) As described below with reference to FIG. 4, thevisual feedback may indicate a position of at least the touch contactpoint of the user of portable electronic device 110 relative to thestrike area of at least one of the virtual command icons in the userinterface. For example, A/V display device 114 may display the visualfeedback on the display along with content, such as A/V content that isgenerated by A/V hub 112 and/or one of the one or moreconsumer-electronic devices 116. In particular, the visual feedback maybe superimposed on or over the A/V content. Moreover, the visualfeedback may be partially transparent so that the A/V content is visibleunderneath the visual feedback when displayed on A/V display device 114.In an exemplary embodiment, the visual feedback may include spatialinformation or graphical information that summarizes the current spatialconfiguration of the user interface (including one or more virtualicons, their functions and/or the associated strike areas) and therelative position of the touch contact point. Note that visual feedbackthat is superimposed on or over the A/V content may be context based.For example, the visual feedback may depend upon or may be based on theelectronic devices that are connected or coupled to A/V hub 112.

In this way, the user may be provide intuitive situational awarenessabout their position in the user interface as the user moves theirfinger or digit (or a stylus) around the surface of the touch-sensitivedisplay while maintaining contact with the touch-sensitive display. Oncethe user has successfully navigated to a desired virtual icon, the usermay release or break contact with the touch-sensitive display, so thatthe user-interface activity information includes an associated commandor instruction for one of the components in system 100.

Although we describe the network environment shown in FIG. 1 as anexample, in alternative embodiments, different numbers or types ofelectronic devices may be present. For example, some embodimentscomprise more or fewer electronic devices. As another example, inanother embodiment, different electronic devices are transmitting and/orreceiving packets or frames. While portable electronic device 110 andA/V hub 112 are illustrated with a single instance of radios 118, inother embodiments portable electronic device 110 and A/V hub 112 (andoptionally one or more of A/V display device 114 and/or the one or moreconsumer-electronic devices 116) may include multiple radios.

FIG. 2 presents embodiments of a flow diagram illustrating method 200for providing visual feedback, which may be performed by an A/V hub,such as A/V hub 112 (FIG. 1). During operation, the A/V hub receives,via an interface circuit in the A/V hub, user-interface activityinformation (operation 210) from a portable electronic device, where theuser-interface activity information is associated with a user interfacedisplayed on a touch-sensitive display in the portable electronicdevice, and the user interface is used to control one of an A/V displaydevice, the A/V hub and a consumer-electronic device, which is otherthan the portable electronic device. Then, the A/V hub generates thevisual feedback based on the user-interface activity information(operation 212). Next, the A/V hub provides the visual feedback to theA/V display device (operation 214) for display on a display in the A/Vdisplay device, where the visual feedback indicates a relative positionof at least a touch contact point of a user of the portable electronicdevice relative to a strike area of at least a virtual command icon inthe user interface.

In these ways, the A/V hub (for example, software executed in anenvironment of the A/V hub, such as an operating system) may facilitatethe visual feedback to the user of the portable electronic device. Thisvisual feedback may allow the user to navigate through the userinterface without the user looking at the user interface. Consequently,method 200 may reduce user errors or mistakes when using the userinterface, which may improve the user experience when using the portableelectronic device.

In some embodiments of method 200 (FIG. 2), there may be additional orfewer operations. For example, the A/V hub may display the visualfeedback on the A/V display device. Moreover, the order of theoperations may be changed, and/or two or more operations may be combinedinto a single operation.

FIG. 3 presents a block diagram illustrating visual feedback on an A/Vdisplay device, such as A/V display device 114. In particular, a display310 in A/V display device 114 may present or display content, such asA/V content 312. In addition, display 310 may present or display thevisual feedback that is received from A/V hub 112 (FIG. 1). As shown inFIG. 1, the visual feedback may include object or graphical information314 that summarizes the current spatial configuration of a userinterface on a touch-sensitive display in a portable electronic device.This graphical information may include representations of: one or morevirtual icons 316, functions 318 of the one or more virtual icons 316and/or strike areas 320 associated with the one or more virtual icons316. In addition, graphical information 314 may indicate a relative orabsolute position of a current touch contact point (TPC) 322 of a userof the portable electronic device with the touch-sensitive display.

When the user activates a given one of the one or more virtual icons 316(such as virtual icon 316-1) by touching the surface of thetouch-sensitive display with a strike area of virtual icon 316-1 (whichmay have a larger area, a smaller area or an equivalent area as virtualicon 316-1, so that the strike area can be between a portion of and theentire area of the touch-sensitive display) and then breaking contactwith the surface of the touch-sensitive display, virtual icon 316-1 maybe shown in bold (e.g., with a different line thickness), in a differentcolor and/or flashed in graphical information 314. Similarly, when theuser deactivates the given one of the one or more virtual icons 316(such as virtual icon 316-1) by touching the surface of thetouch-sensitive display with the strike area of virtual icon 316-1 againand then breaking contact with the surface of the touch-sensitivedisplay, the display of virtual icon 316-1 may revert to its originalformat (such as a default line-segment thickness, no flashing, etc.) ingraphical information 314. Furthermore, as the user moves their touchcontact point on the surface of the touch-sensitive display, theuser-activity information communicated by the portable electronic deviceto the A/V hub may be updated, so that the A/V hub can accordinglyupdate the visual feedback (such as graphical information 314) that isprovided to A/V display device 114 and which is displayed on display310. In this way, A/V display device 114 can dynamically update thevisual feedback presented to the user. In some embodiments, if the userhas not made contact with the surface of the touch-sensitive display formore than a predefined time interval (such as 5, 10 or 30 s), graphicalinformation 314 is no longer displayed on display 310. This fade out (aswell as a subsequent fade in) may be a sharp or sudden transition (suchas in one frame or screen refresh) or made involve a slow transitionover 1-2 s, during which the luminance or greyscale of graphicalinformation 314 is progressively increased (or decreased).

Graphical information 314 may be superimposed on or over A/V content312. In some embodiments, graphical information 314 is partiallytransparent so that A/V content 312 is visible underneath graphicalinformation 314 when displayed on display 310.

Note that a wide variety of display types may be used in A/V displaydevice 114, including: a two-dimensional display, a three-dimensionaldisplay (such as a holographic display or a volumetric display), ahead-mounted display, a retinal-image projector, a heads-up display, acathode ray tube, a liquid-crystal display, a projection display, anelectroluminescent display, a display based on electronic paper, athin-film transistor display, a high-performance addressing display, anorganic light-emitting diode display, a surface-conductionelectronic-emitter display, a laser display, a carbon-nanotube display,a quantum-dot display, an interferometric modulator display, amulti-touch touchscreen (which is sometimes referred to as atouch-sensitive display), and/or a display based on another type ofdisplay technology or physical phenomenon. Thus, display 310 may beportable or at a fixed location.

Embodiments of the feedback technique are further illustrated in FIG. 4,which presents a drawing illustrating communication between portableelectronic device 110 and A/V hub 112. In particular, while a user isusing portable electronic device 110, touch-screen I/O controller 412may provide user-interface activity information 414 to processor 416based on user interaction 410 with touch-sensitive display (TSD 124),such as: the user making or breaking contact with a surface oftouch-sensitive display 124, moving a touch contact point on thesurface, etc. Then, processor 416 may instruct interface circuit 418 tocommunicate user-interface activity information 414 to A/V hub 112.

After interface circuit 420 in A/V hub 112 receives user-interfaceactivity information 414, processor 422 may generate visual feedback 424based on user-interface activity information 414. For example,user-interface activity information 414 may specify a location of atouch contact point on touch-sensitive display 124, the layout andfunctions of one or more virtual command icons in the user interface,and/or may include an instruction or a command to activate or deactivateone or more functions of at least one of the components in system 100(FIG. 1). Based on this user-interface activity information, processor422 may generate instructions for an object or graphical informationthat includes or specifies visual feedback 424, such as a partiallytransparent graphical overlay that can be displayed on A/V displaydevice 114. Note that the graphical information may includedisplay-specific information, such as: a location where the graphicalinformation is to be displayed on a display having a particular type,display size, and/or an aspect ratio or geometry, e.g., an aspect ratioof 16:9 with a display diagonal of at least 50 in. (These values are forpurposes of illustration only, and a wide variety of display sizes,aspect ratios and types may be used in A/V display device 114.)Furthermore, generating visual feedback 424 may involve calculating atwo or three-dimensional model (such as a model of the virtual icons inthe user interface) and/or rendering operations, such as: two orthree-dimensional projection, ray tracing, shading, coloring, texturing,illumination effects, texture mapping, and/or anti-aliasing. In the caseof a three-dimensional display in A/V display device 114, the renderingoperations may include calculating one or more images that include orrepresent: image parallax, motion parallax (based on motion of the userrelative to A/V display device 114) and/or prehension (which may allowthe user to perceive three-dimensional tactile or haptic interactionwith objects).

Processor 422 may then instruct interface circuit 420 (or anotherinterface circuit) to provide visual feedback 424 and/or A/V content 426(which may be generated by A/V hub 112 and/or received from anothercomponent in system 100 in FIG. 1) to A/V display device 114 for displayto the user of portable electronic device 110, thereby assisting theuser in intuitively using the user interface displayed ontouch-sensitive display 124.

We now describe embodiments of an electronic device. FIG. 5 presents ablock diagram illustrating an electronic device 500, such as portableelectronic device 110, A/V hub 112 or A/V display device 114 in FIG. 1.This electronic device includes processing subsystem 510, memorysubsystem 512, and networking subsystem 514. Processing subsystem 510includes one or more devices configured to perform computationaloperations. For example, processing subsystem 510 can include one ormore microprocessors, application-specific integrated circuits (ASICs),microcontrollers, programmable-logic devices, and/or one or more digitalsignal processors (DSPs). One or more of these components in processingsubsystem are sometimes referred to as a ‘control mechanism’ or a‘control circuit.’

Memory subsystem 512 includes one or more devices for storing dataand/or instructions for processing subsystem 510 and networkingsubsystem 514. For example, memory subsystem 512 can include dynamicrandom access memory (DRAM), static random access memory (SRAM), and/orother types of memory. In some embodiments, instructions for processingsubsystem 510 in memory subsystem 512 include: one or more programmodules or sets of instructions (such as program module 522 or operatingsystem 524), which may be executed by processing subsystem 510. Notethat the one or more computer programs may constitute a computer-programmechanism. Moreover, instructions in the various modules in memorysubsystem 512 may be implemented in: a high-level procedural language,an object-oriented programming language, and/or in an assembly ormachine language. Furthermore, the programming language may be compiledor interpreted, e.g., configurable or configured (which may be usedinterchangeably in this discussion), to be executed by processingsubsystem 510.

In addition, memory subsystem 512 can include mechanisms for controllingaccess to the memory. In some embodiments, memory subsystem 512 includesa memory hierarchy that comprises one or more caches coupled to a memoryin electronic device 500. In some of these embodiments, one or more ofthe caches is located in processing subsystem 510.

In some embodiments, memory subsystem 512 is coupled to one or morehigh-capacity mass-storage devices (not shown). For example, memorysubsystem 512 can be coupled to a magnetic or optical drive, asolid-state drive, or another type of mass-storage device. In theseembodiments, memory subsystem 512 can be used by electronic device 500as fast-access storage for often-used data, while the mass-storagedevice is used to store less frequently used data.

Networking subsystem 514 includes one or more devices configured tocouple to and communicate on a wired and/or wireless network (i.e., toperform network operations), including: control logic 516, interfacecircuits 518 and associated antennas 520. (While FIG. 5 includesantennas 520, in some embodiments electronic device 500 includes one ormore nodes, such as nodes 508, e.g., pads, which can be coupled toantennas 520. Thus, electronic device 500 may or may not includeantennas 520.) For example, networking subsystem 514 can include aBluetooth networking system, a cellular networking system (e.g., a 3G/4Gnetwork such as UMTS, LTE, etc.), a universal serial bus (USB)networking system, a networking system based on the standards describedin IEEE 802.11 (e.g., a Wi-Fi networking system), an Ethernet networkingsystem, and/or another networking system. Note that the combination of agiven one of interface circuits 518 and at least one of antennas 520 mayconstitute a radio. In some embodiments, networking subsystem 514includes a wired interface, such as HDMI interface 530.

Networking subsystem 514 includes processors, controllers,radios/antennas, sockets/plugs, and/or other devices used for couplingto, communicating on, and handling data and events for each supportednetworking system. Note that mechanisms used for coupling to,communicating on, and handling data and events on the network for eachnetwork system are sometimes collectively referred to as a ‘networkinterface’ for the network system. Moreover, in some embodiments a‘network’ between the electronic devices does not yet exist. Therefore,electronic device 500 may use the mechanisms in networking subsystem 514for performing simple wireless communication between the electronicdevices, e.g., transmitting advertising or beacon frames and/or scanningfor advertising frames transmitted by other electronic devices asdescribed previously.

Within electronic device 500, processing subsystem 510, memory subsystem512, and networking subsystem 514 are coupled together using bus 528.Bus 528 may include an electrical, optical, and/or electro-opticalconnection that the subsystems can use to communicate commands and dataamong one another. Although only one bus 528 is shown for clarity,different embodiments can include a different number or configuration ofelectrical, optical, and/or electro-optical connections among thesubsystems.

In some embodiments, electronic device 500 includes a display subsystem526 for displaying information on a display (such as the communicationwarning message), which may include a display driver, an I/O controllerand the display, such as a liquid-crystal display, a multi-touchtouchscreen (which is sometimes referred to as a touch-sensitivedisplay), etc.

Electronic device 500 can be (or can be included in) any electronicdevice with at least one network interface. For example, electronicdevice 500 can be (or can be included in): a desktop computer, a laptopcomputer, a subnotebook/netbook, a server, a tablet computer, asmartphone, a cellular telephone, a consumer-electronic device (such asa television, a set-top box, audio equipment, video equipment, etc.), aremote control, a portable computing device, an access point, a router,a switch, communication equipment, test equipment, and/or anotherelectronic device.

Although specific components are used to describe electronic device 500,in alternative embodiments, different components and/or subsystems maybe present in electronic device 500. For example, electronic device 500may include one or more additional processing subsystems, memorysubsystems, networking subsystems, and/or display subsystems. Moreover,while one of antennas 520 is shown coupled to a given one of interfacecircuits 518, there may be multiple antennas coupled to the given one orinstance of interface circuits 518. For example, an instance of a 3×3radio may include three antennas. Additionally, one or more of thesubsystems may not be present in electronic device 500. Furthermore, insome embodiments, electronic device 500 may include one or moreadditional subsystems that are not shown in FIG. 5. Also, althoughseparate subsystems are shown in FIG. 5, in some embodiments, some orall of a given subsystem or component can be integrated into one or moreof the other subsystems or component(s) in electronic device 500. Forexample, in some embodiments program module 522 is included in operatingsystem 524.

Moreover, the circuits and components in electronic device 500 may beimplemented using any combination of analog and/or digital circuitry,including: bipolar, PMOS and/or NMOS gates or transistors. Furthermore,signals in these embodiments may include digital signals that haveapproximately discrete values and/or analog signals that have continuousvalues. Additionally, components and circuits may be single-ended ordifferential, and power supplies may be unipolar or bipolar.

An integrated circuit may implement some or all of the functionality ofnetworking subsystem 514, such as one or more radios. Moreover, theintegrated circuit may include hardware and/or software mechanisms thatare used for transmitting wireless signals from electronic device 500and receiving signals at electronic device 500 from other electronicdevices. Aside from the mechanisms herein described, radios aregenerally known in the art and hence are not described in detail. Ingeneral, networking subsystem 514 and/or the integrated circuit caninclude any number of radios.

In some embodiments, networking subsystem 514 and/or the integratedcircuit include a configuration mechanism (such as one or more hardwareand/or software mechanisms) that configures the radios to transmitand/or receive on a given channel (e.g., a given carrier frequency). Forexample, in some embodiments, the configuration mechanism can be used toswitch the radio from monitoring and/or transmitting on a given channelto monitoring and/or transmitting on a different channel. (Note that‘monitoring’ as used herein comprises receiving signals from otherelectronic devices and possibly performing one or more processingoperations on the received signals, e.g., determining if the receivedsignal comprises an advertising frame, calculating a performance metric,performing spectral analysis, etc.) Furthermore, networking subsystem514 may include at least one port (such as an HDMI port 532) to receiveand/or provide the information in the data stream to A/V display device114 (FIG. 1) and/or one of the one or more consumer-electronic devices116 (FIG. 1).

While a communication protocol compatible with Wi-Fi was used as anillustrative example, the described embodiments may be used in a varietyof network interfaces. Furthermore, while some of the operations in thepreceding embodiments were implemented in hardware or software, ingeneral the operations in the preceding embodiments can be implementedin a wide variety of configurations and architectures. Therefore, someor all of the operations in the preceding embodiments may be performedin hardware, in software or both. For example, at least some of theoperations in the feedback technique may be implemented using programmodule 522, operating system 524 (such as drivers for interface circuits518) and/or in firmware in interface circuits 518. Alternatively oradditionally, at least some of the operations in the feedback techniquemay be implemented in a physical layer, such as hardware in interfacecircuits 518.

Moreover, while the preceding embodiments included a touch-sensitivedisplay in the portable electronic device that the user touches (e.g.,with a finger or digit, or a stylus), in other embodiments the userinterface is display on a display in the portable electronic device andthe user interacts with the user interface without making contact ortouching the surface of the display. For example, the user's interact(s)with the user interface may be determined using time-of-flightmeasurements, motion sensing (such as a Doppler measurement) or anothernon-contact measurement that allows the position, direction of motionand/or speed of the user's finger or digit (or a stylus) relative toposition(s) of one or more virtual command icons to be determined. Inthese embodiments, note that the user may activate a given virtualcommand icon by performing a gesture (such as ‘tapping’ their finger inthe air without making contact with the surface of the display). In someembodiments, the user navigates through the user interface and/oractivates/deactivates functions of one of the components in system 100(FIG. 1) using spoken commands or instructions (i.e., via voicerecognition) and/or based on where they are looking in the visualfeedback displayed on A/V display device 114 in FIG. 1 (e.g., bytracking the user's gaze or where the user is looking).

Furthermore, while A/V hub 112 (FIG. 1) was illustrated as a separatecomponent from A/V display device 114 (FIG. 1), in some embodiments thecomponents are combined into a single component or a single electronicdevice.

While the preceding embodiments illustrated the feedback technique withaudio and video content, in other embodiments the feedback technique isused in the context of an arbitrary type of data or information. Forexample, the feedback technique may be used with home-automation data.In these embodiments, A/V hub 112 (FIG. 1) may facilitate communicationamong and control of a wide variety of electronic devices, includingelectronic devices in addition to or other than entertainment devices.Thus, A/V hub 112 (FIG. 1) and the feedback technique may be used tofacilitate or implement the so-called Internet of things.

In the preceding description, we refer to ‘some embodiments.’ Note that‘some embodiments’ describes a subset of all of the possibleembodiments, but does not always specify the same subset of embodiments.

The foregoing description is intended to enable any person skilled inthe art to make and use the disclosure, and is provided in the contextof a particular application and its requirements. Moreover, theforegoing descriptions of embodiments of the present disclosure havebeen presented for purposes of illustration and description only. Theyare not intended to be exhaustive or to limit the present disclosure tothe forms disclosed. Accordingly, many modifications and variations willbe apparent to practitioners skilled in the art, and the generalprinciples defined herein may be applied to other embodiments andapplications without departing from the spirit and scope of the presentdisclosure. Additionally, the discussion of the preceding embodiments isnot intended to limit the present disclosure. Thus, the presentdisclosure is not intended to be limited to the embodiments shown, butis to be accorded the widest scope consistent with the principles andfeatures disclosed herein.

What is claimed is:
 1. An audio/video (A/V) hub, comprising: an antenna;an interface circuit, coupled to the antenna, which, during operation,communicates with a portable electronic device; and a control circuit,coupled to the interface circuit, which, during operation: receives, viathe interface circuit, user-interface activity information from theportable electronic device, wherein the user-interface activityinformation is associated with a user interface displayed on atouch-sensitive display in the portable electronic device, and whereinthe user interface is used to control one of an audio/video (A/V)display device, the A/V hub and a consumer-electronic device, which isother than the portable electronic device; generates visual feedbackbased on the user-interface activity information; and provides thevisual feedback to the A/V display device for display on a display inthe A/V display device, wherein the visual feedback indicates a positionof at least a touch contact point of a user of the portable electronicdevice relative to a strike area of at least a virtual command icon inthe user interface.
 2. The A/V hub of claim 1, wherein, duringoperation, the interface circuit is communicates with the A/V displaydevice.
 3. The A/V hub of claim 1, wherein the A/V hub further comprisesa second interface circuit, coupled to the processor, which can becoupled to the A/V display device.
 4. The A/V hub of claim 4, whereinthe A/V hub includes a port, coupled to the second interface circuit,compatible with a high-definition multimedia-interface (HDMI) standard;and wherein the A/V hub can be coupled to the A/V display device via theport.
 5. The A/V hub of claim 1, wherein the visual feedback isdisplayed on the display along with A/V content; wherein the visualfeedback is superimposed on the A/V content; and wherein the visualfeedback is partially transparent so that the A/V content is visibleunderneath the visual feedback when displayed on the display.
 6. The A/Vhub of claim 1, wherein, during operation, the control circuit:generates the A/V content; and provides the A/V content to the A/Vdisplay device.
 7. The A/V hub of claim 1, wherein the A/V displaydevice includes a television.
 8. The A/V hub of claim 1, wherein theportable electronic device includes a cellular telephone that functionsas a remote control.
 9. The A/V hub of claim 1, wherein at least thetouch contact point of the user is ongoing.
 10. The A/V hub of claim 1,wherein the user-interface activity information specifies activation ofat least the virtual command icon based on the user releasing at leastthe touch contact point within the strike area.
 11. The A/V hub of claim1, wherein the control circuit comprises: a processor coupled to theinterface circuit; and a memory, coupled to the processor, which storesa program module that, during operation, is executed by the processor,the program module including instructions for: the receiving; thegenerating; and the providing.
 12. A computer-program product for use inconjunction with an audio/video (A/V) hub, the computer-program productcomprising a non-transitory computer-readable storage medium and acomputer-program mechanism embedded therein to provide visual feedback,the computer-program mechanism including: instructions for receiving,via an interface circuit in the A/V hub, user-interface activityinformation from the portable electronic device, wherein theuser-interface activity information is associated with a user interfacedisplayed on a touch-sensitive display in the portable electronicdevice, and wherein the user interface is used to control one of an A/Vdisplay device, the A/V hub and a consumer-electronic device, which isother than the portable electronic device; instructions for generatingthe visual feedback based on the user-interface activity information;and instructions for providing the visual feedback to the A/V displaydevice for display on a display in the A/V display device, wherein thevisual feedback indicates a position of at least a touch contact pointof a user of the portable electronic device relative to a strike area ofat least a virtual command icon in the user interface.
 13. Thecomputer-program product of claim 12, wherein the visual feedback isprovided to the A/V display device via a port that is compatible with ahigh-definition multimedia-interface (HDMI) standard.
 14. Thecomputer-program product of claim 12, wherein the visual feedback isdisplayed on the display along with A/V content; wherein the visualfeedback is superimposed on the A/V content; and wherein the visualfeedback is partially transparent so that the A/V content is visibleunderneath the visual feedback when displayed on the display.
 15. Thecomputer-program product of claim 12, wherein the computer-programmechanism further comprises instructions for: generating the A/Vcontent; and providing the A/V content to the A/V display device. 16.The computer-program product of claim 12, wherein the portableelectronic device includes a cellular telephone that functions as aremote control.
 17. The computer-program product of claim 12, wherein atleast the touch contact point of the user is ongoing.
 18. Thecomputer-program product of claim 12, wherein the user-interfaceactivity information specifies activation of at least the virtualcommand icon based on the user releasing at least the touch contactpoint within the strike area.
 19. An audio/video (A/V) hub-implementedmethod for providing visual feedback, wherein the method comprises:receiving, via an interface circuit in the A/V hub, user-interfaceactivity information from the portable electronic device, wherein theuser-interface activity information is associated with a user interfacedisplayed on a touch-sensitive display in the portable electronicdevice, and wherein the user interface is used to control one of anaudio/video (A/V) display device, the A/V hub and a consumer-electronicdevice, which is other than the portable electronic device; using theA/V hub, generating the visual feedback based on the user-interfaceactivity information; and providing the visual feedback to the A/Vdisplay device for display on a display in the A/V display device,wherein the visual feedback indicates a relative position of at least atouch contact point of a user of the portable electronic device relativeto a strike area of at least a virtual command icon in the userinterface.
 20. The method of claim 19, wherein the portable electronicdevice includes a cellular telephone that functions as a remote control.